Neural Representation Research Articles

Representation of internal speech by single neurons in human supramarginal gyrus.

Speech brain-machine interfaces (BMIs) translate brain signals into words or audio outputs, enabling communication for people having lost their speech abilities due to diseases or injury. While important advances in vocalized, attempted and mimed speech decoding have been achieved, results for internal speech decoding are sparse and have yet to achieve high functionality. Notably, it is still unclear from which brain areas internal speech can be decoded. Here two participants with tetraplegia with implanted microelectrode arrays located in the supramarginal gyrus (SMG) and primary somatosensory cortex (S1) performed internal and vocalized speech of six words and two pseudowords. In both participants, we found significant neural representation of internal and vocalized speech, at the single neuron and population level in the SMG. From recorded population activity in the SMG, the internally spoken and vocalized words were significantly decodable. In an offline analysis, we achieved average decoding accuracies of 55% and 24% for each participant, respectively (chance level 12.5%), and during an online internal speech BMI task, we averaged 79% and 23% accuracy, respectively. Evidence of shared neural representations between internal speech, word reading and vocalized speech processes was found in participant 1. SMG represented words as well as pseudowords, providing evidence for phonetic encoding. Furthermore, our decoder achieved high classification with multiple internal speech strategies (auditory imagination/visual imagination). Activity in S1 was modulated by vocalized but not internal speech in both participants, suggesting no articulator movements of the vocal tract occurred during internal speech production. This work represents a proof-of-concept for a high-performance internal speech BMI.

The tuning of tuning: How adaptation influences single cell information transfer.

Sensory neurons reconstruct the world from action potentials (spikes) impinging on them. To effectively transfer information about the stimulus to the next processing level, a neuron needs to be able to adapt its working range to the properties of the stimulus. Here, we focus on the intrinsic neural properties that influence information transfer in cortical neurons and how tightly their properties need to be tuned to the stimulus statistics for them to be effective. We start by measuring the intrinsic information encoding properties of putative excitatory and inhibitory neurons in L2/3 of the mouse barrel cortex. Excitatory neurons show high thresholds and strong adaptation, making them fire sparsely and resulting in a strong compression of information, whereas inhibitory neurons that favour fast spiking transfer more information. Next, we turn to computational modelling and ask how two properties influence information transfer: 1) spike-frequency adaptation and 2) the shape of the IV-curve. We find that a subthreshold (but not threshold) adaptation, the 'h-current', and a properly tuned leak conductance can increase the information transfer of a neuron, whereas threshold adaptation can increase its working range. Finally, we verify the effect of the IV-curve slope in our experimental recordings and show that excitatory neurons form a more heterogeneous population than inhibitory neurons. These relationships between intrinsic neural features and neural coding that had not been quantified before will aid computational, theoretical and systems neuroscientists in understanding how neuronal populations can alter their coding properties, such as through the impact of neuromodulators. Why the variability of intrinsic properties of excitatory neurons is larger than that of inhibitory ones is an exciting question, for which future research is needed.

Filtering After Shading With Stochastic Texture Filtering

2D texture maps and 3D voxel arrays are widely used to add rich detail to the surfaces and volumes of rendered scenes, and filtered texture lookups are integral to producing high-quality imagery. We show that applying the texture filter after evaluating shading generally gives more accurate imagery than filtering textures before BSDF evaluation, as is current practice. These benefits are not merely theoretical, but are apparent in common cases. We demonstrate that practical and efficient filtering after shading is possible through the use of stochastic sampling of texture filters. Stochastic texture filtering offers additional benefits, including efficient implementation of high-quality texture filters and efficient filtering of textures stored in compressed and sparse data structures, including neural representations. We demonstrate applications in both real-time and offline rendering and show that the additional error from stochastic filtering is minimal. We find that this error is handled well by either spatiotemporal denoising or moderate pixel sampling rates.

Cross-sectional and Longitudinal Age-related Disintegration in Functional Connectivity: Reference Ability Neural Network Cohort.

Some theories of aging have linked age-related cognitive decline to a reduction in distinctiveness of neural processing. Observed age-related correlation increases among disparate cognitive tasks have supported the dedifferentiation hypothesis. We previously showed cross-sectional evidence for age-related correlation decreases instead, supporting an alternative disintegration hypothesis. In the current study, we extended our previous research to a longitudinal sample. We tested 135 participants (20-80 years) at two time points-baseline and 5-year follow-up-on a battery of 12 in-scanner tests, each tapping one of four reference abilities. We performed between-tasks correlations within domain (convergent) and between domain (discriminant) at both the behavioral and neural level, calculating a single measure of construct validity (convergent - discriminant). Cross-sectionally, behavioral construct validity was significantly different from chance at each time point, but longitudinal change was not significant. Analysis by median age split revealed that older adults showed higher behavioral validity, driven by higher discriminant validity (lower between-tasks correlations). Participant-level neural validity decreased over time, with convergent validity consistently greater than discriminant validity; this finding was also observed at the cross-sectional level. In addition, a disproportionate decrease in neural validity with age remained significant after controlling for demographic factors. Factors predicting longitudinal changes in global cognition (mean performance across all 12 tasks) included age, change in neural validity, education, and National Adult Reading Test (premorbid intelligence). Change in neural validity partially mediated the effect of age on change in global cognition. Our findings support the theory of age-related disintegration, linking cognitive decline to changes in neural representations over time.

Multidimensional neural representations of social features during movie viewing.

The social world is dynamic and contextually embedded. Yet, most studies utilize simple stimuli that do not capture the complexity of everyday social episodes. To address this, we implemented a movie viewing paradigm and investigated how the everyday social episodes are processed in the brain. Participants watched one of two movies during an MRI scan. Neural patterns from brain regions involved in social perception, mentalization, action observation, and sensory processing were extracted. Representational similarity analysis results revealed that several labeled social features (including social interaction, mentalization, the actions of others, characters talking about themselves, talking about others, and talking about objects) were represented in superior temporal gyrus (STG) and middle temporal gyrus (MTG). The mentalization feature was also represented throughout the theory of mind network, and characters talking about others engaged the temporoparietal junction (TPJ), suggesting that listeners may spontaneously infer the mental state of those being talked about. In contrast, we did not observe the action representations in frontoparietal regions of the action observation network. The current findings indicate that STG and MTG serve as key regions for social processing, and that listening to characters talk about others elicits spontaneous mental state inference in TPJ during natural movie viewing.

Counterfactual thinking induces different neural patterns of memory modification in anxious individuals

Episodic counterfactual thinking (eCFT) is the process of mentally simulating alternate versions of experiences, which confers new phenomenological properties to the original memory and may be a useful therapeutic target for trait anxiety. However, it remains unclear how the neural representations of a memory change during eCFT. We hypothesized that eCFT-induced memory modification is associated with changes to the neural pattern of a memory primarily within the default mode network, moderated by dispositional anxiety levels. We tested this proposal by examining the representational dynamics of eCFT for 39 participants varying in trait anxiety. During eCFT, lateral parietal regions showed progressively more distinct activity patterns, whereas medial frontal neural activity patterns became more similar to those of the original memory. Neural pattern similarity in many default mode network regions was moderated by trait anxiety, where highly anxious individuals exhibited more generalized representations for upward eCFT (better counterfactual outcomes), but more distinct representations for downward eCFT (worse counterfactual outcomes). Our findings illustrate the efficacy of examining eCFT-based memory modification via neural pattern similarity, as well as the intricate interplay between trait anxiety and eCFT generation.

The frequency-following response in late preterm neonates: a pilot study

IntroductionInfants born very early preterm are at high risk of language delays. However, less is known about the consequences of late prematurity. Hence, the aim of the present study is to characterize the neural encoding of speech sounds in late preterm neonates in comparison with those born at term.MethodsThe speech-evoked frequency-following response (FFR) was recorded to a consonant-vowel stimulus /da/ in 36 neonates in three different groups: 12 preterm neonates [mean gestational age (GA) 36.05 weeks], 12 “early term neonates” (mean GA 38.3 weeks), and “late term neonates” (mean GA 41.01 weeks).ResultsFrom the FFR recordings, a delayed neural response and a weaker stimulus F0 encoding in premature neonates compared to neonates born at term was observed. No differences in the response time onset nor in stimulus F0 encoding were observed between the two groups of neonates born at term. No differences between the three groups were observed in the neural encoding of the stimulus temporal fine structure.DiscussionThese results highlight alterations in the neural encoding of speech sounds related to prematurity, which were present for the stimulus F0 but not for its temporal fine structure.

Neuromorphic place cells

Abstract A neuromorphic SLAM system shows potential for more efficient implementation than its traditional counterpart. At the mean time a neuromorphic model of spatial encoding neurons in silicon could provide insights on the functionality and dynamic between each group of cells. Especially when realistic factors including variations and imperfections on the neural movement encoding are presented to challenge the existing hypothetical models for localization. We demonstrate a mixed-mode implementation for spatial encoding neurons including theta cells, egocentric place cells, and the typical allocentric place cells. Together, they form a biologically plausible network that could reproduce the localization functionality of place cells observed in rodents. The system consists of a theta chip with 128 theta cell units and an FPGA implementing 4 networks for egocentric place cells formation that provides the capability for tracking on a 11 by 11 place cell grid. Experimental results validate the robustness of our model when suffering from as much as 18% deviation, induced by parameter variations in analog circuits, from the mathematical model of theta cells. We provide a model for implementing dynamic neuromorphic SLAM systems for dynamic-scale mapping of cluttered environments, even when subject to significant errors in sensory measurements and real-time analog computation. We also suggest a robust approach for the network topology of spatial cells that can mitigate neural non-uniformity and provides a hypothesis for the function of grid cells and the existence of egocentric place cells.

A feature enhanced EEG compression model using asymmetric encoding-decoding network.

Recently, the demand for wearable devices using electroencephalography (EEG) has increased rapidly in many fields. Due to its volume and computation constraints, wearable devices usually compress and transmit EEG to external devices for analysis. However, current EEG compression algorithms are not tailor-made for wearable devices with limited computing and storage. Firstly, the huge amount of parameters makes it difficult to apply in wearable devices; secondly, it's tricky to learn EEG signals' distribution law due to the low signal-to-noise ratio, which leads to excessive reconstruction error and suboptimal compression performance. Here, a feature enhanced asymmetric encoding-decoding network is proposed. EEG is encoded with a lightweight model, and subsequently decoded with a multi-level feature fusion network by extracting the encoded features deeply and reconstructing the signal through a two-branch structure. On public EEG datasets, motor imagery and event-related potentials, experimental results show that the proposed method has achieved the state of the art compression performance. In addition, the neural representation analysis and the classification performance of the reconstructed EEG signals also show that our method tends to retain more task-related information as the compression ratio increases and retains reliable discriminative information after EEG compression. This paper tailors an asymmetric EEG compression method for wearable devices that achieves state-of-the-art compression performance in a lightweight manner, paving the way for the application of EEG-based wearable devices.

Drift of neural ensembles driven by slow fluctuations of intrinsic excitability.

Representational drift refers to the dynamic nature of neural representations in the brain despite the behavior being seemingly stable. Although drift has been observed in many different brain regions, the mechanisms underlying it are not known. Since intrinsic neural excitability is suggested to play a key role in regulating memory allocation, fluctuations of excitability could bias the reactivation of previously stored memory ensembles and therefore act as a motor for drift. Here, we propose a rate-based plastic recurrent neural network with slow fluctuations of intrinsic excitability. We first show that subsequent reactivations of a neural ensemble can lead to drift of this ensemble. The model predicts that drift is induced by co-activation of previously active neurons along with neurons with high excitability which leads to remodeling of the recurrent weights. Consistent with previous experimental works, the drifting ensemble is informative about its temporal history. Crucially, we show that the gradual nature of the drift is necessary for decoding temporal information from the activity of the ensemble. Finally, we show that the memory is preserved and can be decoded by an output neuron having plastic synapses with the main region.

Drift of neural ensembles driven by slow fluctuations of intrinsic excitability

Representational drift refers to the dynamic nature of neural representations in the brain despite the behavior being seemingly stable. Although drift has been observed in many different brain regions, the mechanisms underlying it are not known. Since intrinsic neural excitability is suggested to play a key role in regulating memory allocation, fluctuations of excitability could bias the reactivation of previously stored memory ensembles and therefore act as a motor for drift. Here, we propose a rate-based plastic recurrent neural network with slow fluctuations of intrinsic excitability. We first show that subsequent reactivations of a neural ensemble can lead to drift of this ensemble. The model predicts that drift is induced by co-activation of previously active neurons along with neurons with high excitability which leads to remodeling of the recurrent weights. Consistent with previous experimental works, the drifting ensemble is informative about its temporal history. Crucially, we show that the gradual nature of the drift is necessary for decoding temporal information from the activity of the ensemble. Finally, we show that the memory is preserved and can be decoded by an output neuron having plastic synapses with the main region.

Loneliness corresponds with neural representations and language use that deviate from shared cultural perceptions

The word zeitgeist refers to common perceptions shared in a given culture. Meanwhile, a defining feature of loneliness is feeling that one’s views are not shared with others. Does loneliness correspond with deviating from the zeitgeist? Across two independent brain imaging datasets, lonely participants’ neural representations of well-known celebrities strayed from group-consensus neural representations in the medial prefrontal cortex—a region that encodes and retrieves social knowledge (Studies 1 A/1B: N = 40 each). Because communication fosters social connection by creating shared reality, we next asked whether lonelier participants’ communication about well-known celebrities also deviates from the zeitgeist. Indeed, when a strong group consensus exists, lonelier individuals use idiosyncratic language to describe well-known celebrities (Study 2: N = 923). Collectively, results support lonely individuals’ feeling that their views are not shared. This suggests loneliness may not only reflect impoverished relationships with specific individuals, but also feelings of disconnection from prevalently shared views of contemporary culture.

Brain2GAN: Feature-disentangled neural encoding and decoding of visual perception in the primate brain.

A challenging goal of neural coding is to characterize the neural representations underlying visual perception. To this end, multi-unit activity (MUA) of macaque visual cortex was recorded in a passive fixation task upon presentation of faces and natural images. We analyzed the relationship between MUA and latent representations of state-of-the-art deep generative models, including the conventional and feature-disentangled representations of generative adversarial networks (GANs) (i.e., z- and w-latents of StyleGAN, respectively) and language-contrastive representations of latent diffusion networks (i.e., CLIP-latents of Stable Diffusion). A mass univariate neural encoding analysis of the latent representations showed that feature-disentangled w representations outperform both z and CLIP representations in explaining neural responses. Further, w-latent features were found to be positioned at the higher end of the complexity gradient which indicates that they capture visual information relevant to high-level neural activity. Subsequently, a multivariate neural decoding analysis of the feature-disentangled representations resulted in state-of-the-art spatiotemporal reconstructions of visual perception. Taken together, our results not only highlight the important role of feature-disentanglement in shaping high-level neural representations underlying visual perception but also serve as an important benchmark for the future of neural coding.

Neurometric amplitude modulation detection in the inferior colliculus of Young and Aged rats

Amplitude modulation is an important acoustic cue for sound discrimination, and humans and animals are able to detect small modulation depths behaviorally. In the inferior colliculus (IC), both firing rate and phase-locking may be used to detect amplitude modulation. How neural representations that detect modulation change with age are poorly understood, including the extent to which age-related changes may be attributed to the inherited properties of ascending inputs to IC neurons. Here, simultaneous measures of local field potentials (LFPs) and single-unit responses were made from the inferior colliculus of Young and Aged rats using both noise and tone carriers in response to sinusoidally amplitude-modulated sounds of varying depths. We found that Young units had higher firing rates than Aged for noise carriers, whereas Aged units had higher phase-locking (vector strength), especially for tone carriers. Sustained LFPs were larger in Young animals for modulation frequencies 8–16 Hz and comparable at higher modulation frequencies. Onset LFP amplitudes were much larger in Young animals and were correlated with the evoked firing rates, while LFP onset latencies were shorter in Aged animals. Unit neurometric thresholds by synchrony or firing rate measures did not differ significantly across age and were comparable to behavioral thresholds in previous studies whereas LFP thresholds were lower than behavior.

Eye movement intervention facilitates concurrent perception and memory processing.

A widely used psychotherapeutic treatment for post-traumatic stress disorder (PTSD) involves performing bilateral eye movement (EM) during trauma memory retrieval. However, how this treatment-described as eye movement desensitization and reprocessing (EMDR)-alleviates trauma-related symptoms is unclear. While conventional theories suggest that bilateral EM interferes with concurrently retrieved trauma memories by taxing the limited working memory resources, here, we propose that bilateral EM actually facilitates information processing. In two EEG experiments, we replicated the bilateral EM procedure of EMDR, having participants engaging in continuous bilateral EM or receiving bilateral sensory stimulation (BS) as a control while retrieving short- or long-term memory. During EM or BS, we presented bystander images or memory cues to probe neural representations of perceptual and memory information. Multivariate pattern analysis of the EEG signals revealed that bilateral EM enhanced neural representations of simultaneously processed perceptual and memory information. This enhancement was accompanied by heightened visual responses and increased neural excitability in the occipital region. Furthermore, bilateral EM increased information transmission from the occipital to the frontoparietal region, indicating facilitated information transition from low-level perceptual representation to high-level memory representation. These findings argue for theories that emphasize information facilitation rather than disruption in the EMDR treatment.

Dynamic CBCT imaging using prior model-free spatiotemporal implicit neural representation (PMF-STINR).

Dynamic cone-beam computed tomography (CBCT) can capture high-spatial-resolution, time-varying images for motion monitoring, patient setup, and adaptive planning of radiotherapy. However, dynamic CBCT reconstruction is an extremely ill-posed spatiotemporal inverse problem, as each CBCT volume in the dynamic sequence is only captured by one or a few X-ray projections, due to the slow gantry rotation speed and the fast anatomical motion (e.g., breathing). 
Approach: We developed a machine learning-based technique, prior-model-free spatiotemporal implicit neural representation (PMF-STINR), to reconstruct dynamic CBCTs from sequentially acquired X-ray projections. PMF-STINR employs a joint image reconstruction and registration approach to address the under-sampling challenge, enabling dynamic CBCT reconstruction from singular X-ray projections. Specifically, PMF-STINR uses spatial implicit neural representations to reconstruct a reference CBCT volume, and it applies temporal INR to represent the intra-scan dynamic motion with respect to the reference CBCT to yield dynamic CBCTs. PMF-STINR couples the temporal INR with a learning-based B-spline motion model to capture time-varying deformable motion during the reconstruction. Compared with the previous methods, the spatial INR, the temporal INR, and the B-spline model of PMF-STINR are all learned on the fly during reconstruction in a one-shot fashion, without using any patient-specific prior knowledge or motion sorting/binning. 
Main results: PMF-STINR was evaluated via digital phantom simulations, physical phantom measurements, and a multi-institutional patient dataset featuring various imaging protocols (half-fan/full-fan, full sampling/sparse sampling, different energy and mAs settings, etc.). The results showed that the one-shot learning-based PMF-STINR can accurately and robustly reconstruct dynamic CBCTs and capture highly irregular motion with high temporal (~0.1s) resolution and sub-millimeter accuracy.
Significance: PMF-STINR can reconstruct dynamic CBCTs and solve the intra-scan motion problem from conventional 3D CBCT scans without using any prior anatomical/motion model or motion sorting/binning. It can be a promising tool for motion management by offering richer motion information than traditional 4D-CBCTs.

Atypical neural encoding of faces in individuals with autism spectrum disorder.

Individuals with autism spectrum disorder (ASD) experience pervasive difficulties in processing social information from faces. However, the behavioral and neural mechanisms underlying social trait judgments of faces in ASD remain largely unclear. Here, we comprehensively addressed this question by employing functional neuroimaging and parametrically generated faces that vary in facial trustworthiness and dominance. Behaviorally, participants with ASD exhibited reduced specificity but increased inter-rater variability in social trait judgments. Neurally, participants with ASD showed hypo-activation across broad face-processing areas. Multivariate analysis based on trial-by-trial face responses could discriminate participant groups in the majority of the face-processing areas. Encoding social traits in ASD engaged vastly different face-processing areas compared to controls, and encoding different social traits engaged different brain areas. Interestingly, the idiosyncratic brain areas encoding social traits in ASD were still flexible and context-dependent, similar to neurotypicals. Additionally, participants with ASD also showed an altered encoding of facial saliency features in the eyes and mouth. Together, our results provide a comprehensive understanding of the neural mechanisms underlying social trait judgments in ASD.

A Hybrid Neural Coding Approach for Pattern Recognition With Spiking Neural Networks.

Recently, brain-inspired spiking neural networks (SNNs) have demonstrated promising capabilities in solving pattern recognition tasks. However, these SNNs are grounded on homogeneous neurons that utilize a uniform neural coding for information representation. Given that each neural coding scheme possesses its own merits and drawbacks, these SNNs encounter challenges in achieving optimal performance such as accuracy, response time, efficiency, and robustness, all of which are crucial for practical applications. In this study, we argue that SNN architectures should be holistically designed to incorporate heterogeneous coding schemes. As an initial exploration in this direction, we propose a hybrid neural coding and learning framework, which encompasses a neural coding zoo with diverse neural coding schemes discovered in neuroscience. Additionally, it incorporates a flexible neural coding assignment strategy to accommodate task-specific requirements, along with novel layer-wise learning methods to effectively implement hybrid coding SNNs. We demonstrate the superiority of the proposed framework on image classification and sound localization tasks. Specifically, the proposed hybrid coding SNNs achieve comparable accuracy to state-of-the-art SNNs, while exhibiting significantly reduced inference latency and energy consumption, as well as high noise robustness. This study yields valuable insights into hybrid neural coding designs, paving the way for developing high-performance neuromorphic systems.

Mixed selectivity: Cellular computations for complexity.

The property of mixed selectivity has been discussed at a computational level and offers a strategy to maximize computational power by adding versatility to the functional role of each neuron. Here, we offer a biologically grounded implementational-level mechanistic explanation for mixed selectivity in neural circuits. We define pure, linear, and nonlinear mixed selectivity and discuss how these response properties can be obtained in simple neural circuits. Neurons that respond to multiple, statistically independent variables display mixed selectivity. If their activity can be expressed as a weighted sum, then they exhibit linear mixed selectivity; otherwise, they exhibit nonlinear mixed selectivity. Neural representations based on diverse nonlinear mixed selectivity are high dimensional; hence, they confer enormous flexibility to a simple downstream readout neural circuit. However, a simple neural circuit cannot possibly encode all possible mixtures of variables simultaneously, as this would require a combinatorially large number of mixed selectivity neurons. Gating mechanisms like oscillations and neuromodulation can solve this problem by dynamically selecting which variables are mixed and transmitted to the readout.

Spectral-Wise Implicit Neural Representation for Hyperspectral Image Reconstruction

Spectral-Wise Implicit Neural Representation for Hyperspectral Image Reconstruction